Imaging apparatus, imaging method, reproducing apparatus and reproducing method, and recording medium

ABSTRACT

A sound is recorded when shooting a still image, an effect is applied on the still image in accordance with an analytical result of the recorded sound, and the sound and the effect-applied still image are recorded in association with each other, thereby allowing recording of the image having reality in addition to the sound. The still image and the sound are reproduced in association with each other, thereby allowing a viewer to feel an ambience and reality close to a memory that the photographer felt. Thus, image which easily reminds the situation and feeling during shooting can be recorded with a sound.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique for executing various kindsof processings on captured images according to a situation at which asubject is photographed.

2. Description of the Related Art

An imaging apparatus according to Japanese Patent Application Laid-OpenNo. 2006-203860 includes: an imaging unit for capturing an image of asubject; a recording unit for recording a sound surrounding the imagingunit; a threshold sound volume storage unit for storing a specifiedthreshold sound volume; a sound extraction unit for extracting a soundwithin a part of period including a sound having the volume larger thanthe threshold sound volume stored in the threshold sound volume storageunit among the sound recorded by the recording unit; a data storage unitfor associating each of a plurality of images captured by the imagingunit with each of a plurality of sounds extracted by the soundextraction unit in the order of imaging and recording, and for storingthe same; and a data output unit for outputting the captured image andthe sound, which the data storage unit has stored in association witheach other, in synchronization with each other.

In a camera according to Japanese Patent Application Laid-Open No.2007-221498, music and a sample image suitable for the music arerecorded in a first memory. A similarity detecting section of a similarimage selecting unit refers to first auxiliary data or second auxiliarydata of the sample image and detects a photographed image that resemblesthe sample image among photographed images recorded in a second memory.When a proper photographed image can not be detected, the sample imageis selected. When reproducing music and image, a selected similar imageis reproduced in place of the sample image in accordance with theprogress of the music.

A camera according to Japanese Patent Application Laid-Open No.2007-235432 records sounds inputted by a sound input device according tocontinuous shooting control on a plurality of still images (A-D) by acontinuous shooting control device; and stores and manages, among theplurality of continuously photographed still images (A-D), the stillimage (B) which has been photographed when a sound level determiningdevice determined that an actual sound exceeding a certain level wasinputted, and sound data recorded by a sound recording device inassociation with each other. Even for reproducing individual still imagedata with sounds, the camera can reproduce and output the recorded sounddata only when the still image (B) photographed at the timing when thesounds was actually produced is reproduced and displayed, thusreproducing individual still images with sounds without strange feeling.

SUMMARY OF THE INVENTION

The conventional imaging apparatus such as a digital still camerarecords a momentary image, and records images and sounds in a certaintime period. One of the most important purposes of a user using thecamera is “recording a memory.” However, since a human memory is made upof not only images at the situation but also various senses such assounds, odors, touches and tastes, it is difficult for the user tovividly recollect the reality only with a captured image.

In view of this point, according to Japanese Patent ApplicationLaid-Open No. 2006-203860, a sound is extracted in shooting inaccordance with the sound volume of sound data to be reproduced onconcurrent recording/reproducing of still images and sounds. However,the feature of the sound is not utilized for reproducing the image atall.

According to Japanese Patent Application Laid-Open No. 2007-221498,images similar to a sample image are retrieved among previously-recordedcaptured images, music data and dedicated sample images, and theretrieved images are reproduced in synchronization with the reproductionof the music. However, since the previously recorded music is used, themusic does not always match with the shooting situation of the image,and it is difficult to represent the reality during the shooting.Furthermore, whenever the same sample image is selected, the same imageand music is reproduced, expressing unvarying tiredness.

In Japanese Patent Application Laid-Open No. 2007-235432,Continuous-shot images and sounds are recorded. When the sound is louderor equal to a certain threshold, the sound and the image captured at thesame time are associated with each other and recorded. However, a partwhere the sound becomes loud is not always appropriate for the stillimage expressing the reality at the shooting situation. Instead, it ispreferable that a user decides the shooting timing.

Accordingly, the present invention aims to record, with a sound, animage which easily reminds the situation and feeling during shooting.

An imaging apparatus according to an aspect of the present inventioncomprises: an imaging unit which converts an electric signal into imagedata and outputs the image data, the electric signal being acquired byan image pickup element photoelectrically converting light from asubject image-formed through an imaging lens on a photosensitive surfaceof the image pickup element; a sound acquiring unit which converts asound around the subject into sound data and outputs the sound data; asound analyzing unit which analyzes a feature of the sound based on thesound data outputted from the sound acquiring unit; an effectapplication unit which applies an effect on the image data outputtedfrom the imaging unit, the effect being image processing in accordancewith the feature of the sound analyzed by the sound analyzing unit; anda recording unit which associates the image data generated by applyingthe effect by the effect application unit with the sound data outputtedfrom the sound acquiring unit and records the image data and the sounddata on a prescribed recording medium.

The effect can include image processing which changes at least one of ahue, sharpness, contrast and blurredness of the image.

The sound analyzing unit can extract a sound volume, a frequency, or atime-series variation thereof as a feature of the sound, compares theextracted feature of the sound with a previously-stored feature pattern,and applies the effect which is previously associated with a featurepattern matching the extracted feature of the sound.

The imaging apparatus can further comprise a sound acquisition controlunit which controls a timing of a start, a stop or a termination ofconversion and output of the sound data by the sound acquiring unit inaccordance with the feature of the sound analyzed by the sound analyzingunit.

The sound acquisition control unit can start outputting the sound datawhen detecting that the volume of the sound becomes larger than or equalto a prescribed value, terminates outputting the sound data whenoutputting the sound data over a prescribed first time period, and stopsoutputting the sound data when the volume of the sound falls equal to orbelow the prescribed value and the falling continues over a prescribedsecond time period.

The imaging apparatus can further comprise a moving image creating unitwhich creates image frames of a moving image by applying a plurality ofdifferent effects on the image data outputted from the imaging unit, andcreates moving image data based on the image frames. And, the recordingunit associates the moving image data created by the moving imagecreating unit and the sound data with each other and records theassociated data.

The imaging apparatus can further comprise a continuous shooting controlunit which controls the imaging unit to output a prescribed count ofimage data at prescribed time intervals. And, the effect applicationunit executes image processing on the prescribed count of image dataoutputted from the imaging unit using the image data and another imagedata captured before and/or after the image data in accordance withcontrol of the continuous control unit.

The imaging apparatus can further comprise a sound editing unit whichedits a content of the sound data from the sound acquiring unit inaccordance with the feature of the sound analyzed by the sound analyzingunit.

The sound editing unit can perform at least one of extracting a specificfrequency from the sound data, eliminating noise, extracting a sound ofa specific person, and cutting off sound data whose sound volume doesnot reach a prescribed sound volume.

The imaging apparatus can further comprise a detection unit whichdetects a physical status of the imaging apparatus. And, the effectapplication unit applies the effect in accordance with the physicalstatus detected by the detection unit.

The effect application unit can apply a specific effect in accordancewith a specific physical status detected by the detection unit.

The effect application unit can at least one of a magnification ratio, azooming speed, an amount of shift, a shift rate, turning ratio, andturning speed in the processing of the effect on the image in accordancewith a variation of the volume of the sound or a rate of variation ofthe sound.

The present invention also includes a reproducing apparatus reproducingimage data and sound data in association with each other, the image dataand sound data recorded on the prescribed recording medium by theimaging apparatus.

An imaging method according to another aspect of the present inventioncomprises: converting an electric signal into image data and outputtingthe image data, the electric signal being acquired by an image pickupelement photoelectrically converting light from a subject image-formedthrough an imaging lens on a photosensitive surface of the image pickupelement; converting a sound around the subject into sound data andoutputting the sound data; analyzing a feature of the sound based on theoutputted sound data; applying an effect on the outputted image data,the effect being image processing in accordance with the analyzedfeature of the sound; and associating the image data generated byapplying the effect with the outputted sound data and recording theimage data and the sound data on a prescribed recording medium.

The present invention also includes a reproducing method reproducing theimage data and the sound data in association with each other, the imagedata and sound data recorded on the prescribed recording medium by theimaging method.

The present invention also includes a recording medium on which aprogram is stored, the program comprising computer-executableinstructions of: acquiring image data and sound data recordedconcurrently with the image data; analyzing a feature of the sound basedon the sound data; applying an effect on the image data, the effectbeing image processing in accordance with the analyzed feature of thesound; and reproducing image data generated by applying the effect andthe sound data in association with each other.

Accordingly, a sound is recorded when shooting a still image, an effectis applied on the still image in accordance with an analytical result ofthe recorded sound, and the sound and the effect-applied still image arerecorded in association with each other, thereby allowing recording ofthe image having reality in addition to the sound. Further, the stillimage and the sound are reproduced in association with each other,thereby allowing a viewer to feel an ambience and reality close to amemory that the photographer felt.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of the front face of a digital camera;

FIG. 2 is a perspective view of the rear face of the digital camera;

FIG. 3 is a block diagram showing the internal configuration of thedigital camera;

FIG. 4 is a flow chart of shooting processing according to a firstexemplary embodiment;

FIGS. 5A to 5D show examples of an original image and images on whicheffects have been applied acquired by the shooting processing accordingto the first exemplary embodiment;

FIGS. 6A to 6D show an example of association between an analyticalresult on a sound and an effect processing to be executed by the imageeffector 152 in response to notification of the analytical result on thesound;

FIG. 7 is a flow chart of a shooting processing according to a secondexemplary embodiment;

FIG. 8 shows an example of a timing chart concerning start/terminationof sound recording performed according to sound recording processing;

FIG. 9 shows an example of frames of a moving image;

FIG. 10 shows examples of an effect processing of zooming-in;

FIG. 11 shows an example of an effect processing on continuous images;

FIGS. 12A to 12C schematically show states where an original sound hasbeen edited;

FIG. 13 schematically shows time-series variations in shake amount;

FIG. 14 shows an example where an effect processing is applied on anoriginal image based on a parameter according to shake amount;

FIG. 15 schematically shows time-series variations when a camera body ismomentarily shaken; and

FIGS. 16A to 16C show examples of an effect processing when a camerabody is momentarily shaken.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First ExemplaryEmbodiment

A best mode of preferred embodiments realized in an imaging apparatusaccording to the present invention will be described below withreference to the accompanying drawings.

FIGS. 1 and 2 show perspective views of the front and rear faces,respectively, of a digital camera 10 to which the present invention isapplied.

As shown in FIG. 1, on the front face of a camera body 12, there aredisposed a lens 14, an electronic flash 16, a microphone 17, a viewfinder 18, a self-timer lamp 20 and so forth, and a shutter releasebutton 26, a power switch button 28 and so forth are disposed on the topface. The microphone 17 records sounds simultaneously when moving imagesor still images are acquired. A loudspeaker 35 is disposed on a sideface of the camera body 12. The loudspeaker 35 outputs a shutter clickwhen the shutter release button 26 is pressed, and outputs a soundassociated with images when the images are reproduced.

As shown in FIG. 2, on the rear face of the camera body 12, there aredisposed a monitor 30, a view finder eyepiece 32, a flash lightingbutton 34, a macro button 36, a zooming lever 38, a display button 40, aBACK button 42, a menu/OK button 44, a cross button 48, a mode selectorswitch 50 and so forth.

The shutter release button 26 is configured of a two-step stroke switchwhich permits half pressing and full pressing. Half pressing of thisshutter release button 26 actuates the AE/AF (Auto Exposure/Auto Focus)function of the camera, and an image is photographed when the button isfull-pressed.

The monitor 30 includes a liquid crystal display capable of colordisplaying. The monitor 30 is used as a display screen for both recordedpictures and user interfacing. The monitor 30 also serves as anelectronic view finder for confirming the angle of view when an image isphotographed. The display button 40 functions as a button to instructchanging of the displayed content on the monitor 30.

The electronic flash button 34 functions as a button which instructs tochange over the electronic flash mode. Each time the electronic flashbutton 34 is pressed, the electronic flash mode in use when an image isphotographed is changed over from “Auto Electronic Flash” to “Anti-RedEye Electronic Flash”, “Forced Electronic Flash Lighting”, “SlowSynchronization” and “Prohibit Electronic Flash Lighting” in thatsequence.

The macro button 36 functions as a button to instruct ON/OFF switchingof the macro function. Each time the macro button 36 is pressed, themacro function to be used when an image is photographed is turned on oroff.

The zooming lever 38 functions as a lever to instruct zooming (in orout) on the subject whose image is to be photographed, and alsofunctions as a lever to instruct zooming (enlarging or reducing) thedisplayed image when reproducing. The zooming lever 38 is swingable upand down. Turning it upward results in zooming in, while turning itdownward results in zooming out.

The menu/OK button 44 functions as a button to instruct transition froma normal screen to a menu screen in each mode (menu button), and alsofunctions as a button to make definite a choice and to instruct itsexecution (OK button). The BACK button 42 functions as a button toinstruct cancellation of an input operation or the like.

The cross button 48 functions as a button to enter instructions in fourdirections, up and down, right and left, and is used for selecting anyitem on a menu screen or other similar purposes.

The mode selector switch 50 functions as a switch which instructs tochange modes, and is slidably provided between “Shooting Position” and“Playback Position”. When the mode selector switch 50 is turned to its“Shooting Position”, the digital camera 10 is set to the “ShootingMode”, or when it is turned to its “Playback Position”, the camera isset to the “Playback Mode”.

FIG. 3 is a block diagram schematically illustrating the internalconfiguration of the digital camera 10 shown in FIGS. 1 and 2.

The overall operation of the digital camera 10 is under the integratedcontrol of a central processing unit (CPU) 110. The CPU 110 controls thecamera system under prescribed programs on the basis of inputinformation from an operating unit 112 (including the shutter releasebutton 26, power switch button 28, electronic flash button 34, macrobutton 36, zooming lever 38, display button 40, BACK button 42, menu/OKbutton 44, cross button 48 and mode selector switch 50).

A ROM 116 connected to the CPU 110 via a bus 114 stores programsexecuted by the CPU 110 and various data and the like necessary forcontrol, and an EEPROM 118 stores various items of setting informationregarding the operation of the digital camera 10, including foldermanagement information. A memory (SDRAM) 120 is used not only as thearea for arithmetic operations by the CPU 110 but also as that fortemporary storage of image data and sound data. A VRAM 122 is used as atemporary storage area dedicated to image data.

The digital camera 10 is enabled to photograph an image when the modeselector switch 50 is set to the shooting mode, and power is thensupplied to the imaging unit including a charge coupled device (CCD)124.

Light having passed the lens 14 forms an image on the light receivingface of the CCD 124 via an aperture stop 15. Many photodiodes (lightreceiving elements) are two-dimensionally arranged on the lightreceiving face of the CCD 124, and primary color filters of red (R),green (G) and blue (B) matching the photodiodes are arranged in aprescribed arrangement structure (such as Bayer, G stripe or else). TheCCD 124 has an electronic shutter function to control the duration ofelectric charge accumulation in the photodiodes (shutter speed), and theCPU 110 controls the duration of electric charge accumulation in the CCD124 via a timing generator 126.

The image of the subject formed on the light receiving face of the CCD124 is converted by the photodiodes into signal charges matching thequantity of incident light. The signal charges accumulated in therespective photodiodes are successively read out in accordance with aninstruction from the CPU 110 as voltage signals (image signals) matchingthe signal charges on the basis of drive pulses provided by the timinggenerator 126.

The image signals outputted from the CCD 124 are delivered to an analogprocessing unit (CDS/AMP) 128, where R, G and B signals for individualpixels, after being subjected to sampling hold (correlated doublesampling processing), are amplified and fed to an A/D converter 130.

The A/D converter 130 converts R, G and B signals outputted from theanalog processing unit 128 into digital R, G and B signals. The digitalR, G and B signals outputted from the A/D converter 130 are stored intothe memory 120 via an image input controller 132.

An image signal processing circuit 134 processes R, G and B signalsstored in the memory 120 in accordance with instructions from the CPU110. More specifically, the image signal processing circuit 134functions as image processing device containing a synchronizing circuit(a processing circuit for interpolating spatial discrepancies of colorsignals entailed by the color filter arrangement of one-charged CCD andconverting the color signals into simultaneous signals), a white balancecorrection circuit, a gamma correction circuit, a contour correctioncircuit and a luminance/color differential signal generating circuit,and performs a prescribed signal processing in accordance withinstructions from the CPU 110 while utilizing the memory 120. R, G and Bsignals entered into the image signal processing circuit 134 areconverted into luminance signals (Y signals) and color differentialsignals (Cr and Cb signals) by the image signal processing circuit 134and, after going through a prescribed processing including gammacorrection, are stored into the VRAM 122.

When a captured image is to be displayed on the monitor 30, the image isdelivered from the VRAM 122 to a video encoder 136 via a character MIXcircuit 135. The video encoder 136 converts the entered image data intosignals of a prescribed formula for the displaying purpose (e.g. colorcomposite video signals of the NTSC (National Television SystemCommittee) system) and supplies the converted signals to the monitor 30.Thus, the image captured by the CCD 124 is displayed on the monitor 30.

By periodically taking in image signals from the CCD 124, replacingperiodically image data in the VRAM 122 with image data generated fromthose image signals, and supplying those replacing image data to themonitor 30, an image captured by the CCD 124 is displayed on a real timebasis. The photographer can confirm the angle of view by looking at thispicture displayed on the monitor 30 on a real time basis (throughimage).

A gyro sensor 86 activates when the operational mode of the digitalcamera 10 is the still image shooting mode or the moving image shootingmode, detects rotation of the camera body 12 and supplies the detectedrotation as a rotational signal to a shake correction processing circuit87.

The shake correction processing circuit 87 computes the magnitude ofvibration of the camera body 12 on the basis of the rotational signalwhen the operational mode is the still image shooting mode or the movingimage shooting mode, supplies a correction signal to cancel thevibration to a movable stage (not shown) on which the CCD 124 ismounted, and corrects shakes in the through image and the actual image(image to be actually photographed) by moving the CCD 124 together withthe movable stage.

The character MIX circuit 135 combines prescribed characters andsymbolic data supplied from the memory 120 with the image data suppliedfrom the VRAM 122 in accordance with commands from the CPU 110, andsupplies the synthesized image data to the video encoder 136. Thus, thethrough image on which information including characters and symbols aresuperposed is displayed.

When the operational mode is the still image shooting mode or the movingimage shooting mode, an analog sound signal acquired by the microphone17 is amplified by a sound data processing circuit 149, and thenconverted into digital data and data processing, for instancecompression in a prescribed compression recording system (MP3 etc.), isexecuted on the converted data. The processed data is temporarily storedon the SDRAM 120 and then recorded on a recording medium 150 togetherwith the acquired image. Recorded sound data stored on the SDRAM 120 orthe recording medium 150, and various kinds of notifying sound dataincluding an operational sound, a shutter click sound, a focusing soundof AF and so forth stored previously on the EEPROM 118 are subjected todata processing including decoding, analog sound signal conversion andamplification by the sound data processing circuit 149, and thensupplied to the loudspeaker 35. The loudspeaker 35 outputs the sound inaccordance with the analog sound signal supplied by the sound dataprocessing circuit 149.

Photographing is performed by pressing down the shutter release button26. When the shutter release button 26 is half-pressed first, an S1-onsignal is entered into the CPU 110, and the CPU 110 starts an AE/AFprocessing.

First, the image signals taken in from the CCD 124 via the image inputcontroller 132 are entered into an AF detecting circuit 138 and anAE/AWB (Auto Exposure/Auto White Balance) detecting circuit 140.

The AE/AWB detecting circuit 140, which includes a circuit which divideseach frame into a plurality of areas (e.g. 16×16) and cumulatively addsthe R, G and B signals in each divided area, provides the cumulativesums to the CPU 110. The CPU 110 detects the brightness of the subject(subject luminance) on the basis of the cumulative sums acquired fromthe AE/AWB detecting circuit 140, and computes the exposure value(photographic EV) suitable for the photographing. Then the CPU 110determines the aperture stop value and the shutter speed from thefigured-out photographic EV and prescribed program lines, andaccordingly controls the electronic shutter release of the CCD 124 andan aperture drive unit 142 to achieve an appropriate quantity ofexposure.

When the white balance is automatically adjusted, the AE/AWB detectingcircuit 140 computes the average cumulative sums of the R, G and Bsignals separately for each color and for each divided area, andprovides the computed results to the CPU 110. The CPU 110 calculates theR/G and B/G ratios in each divided area from the acquired cumulativesums of R, B and G, and determines the type of light source on the basisof the distribution of the calculated R/G and B/G values in the colorspaces of R/G and B/G and so on. Then according to the white balanceadjustment value suitable for the determined type of light source, thegains values of the white balance adjusting circuit for the R, G and Bsignals (white balance correction values) are controlled, for instanceto make the value of each ratio approximately 1 (i.e. the cumulative RGBratio per frame R:G:B≅1:1:1), and signals on the each color channel arecorrected.

The AF detecting circuit 138 includes a high pass filter which passesonly the high frequency component of G signals, an absolutizing unit, anAF area extracting unit for cutting out signals in a prescribed focusarea (e.g. the central area of the frame), and an accumulating unit forcumulatively adding absolute value data in the AF area, and the data ofcumulative sums calculated by the AF detecting circuit 138 is notifiedto the CPU 110. The CPU 110, while controlling a focusing lens driveunit 144A to shift a focusing lens 14A, computes focus evaluation values(AF evaluation values) at a plurality of AF detection points, anddetermines the lens position where the evaluation value is the localmaximum as the in-focus position. Then the CPU 110 so controls thefocusing lens drive unit 144A so as to shift the focusing lens 14A tothe in-focus position so determined.

Thus, in response to the half pressing of the shutter release button 26,an AE/AF processing is performed.

The photographer manipulates as required the zooming lever 38 to zoomthe lens 14 to adjust the angle of view. When the zooming lever 38 isturned upward (in the telescopic direction) and a zoom-in signal isentered into the CPU 110, the CPU 110 in response to this signal drivesa zoom lens drive unit 144B to shift a zoom lens 14B in the telescopicdirection. When the zooming lever 38 is turned downward (in the wideangle direction) and a zoom-out signal is entered into the CPU 110, theCPU 110 in response to this signal drives the zoom lens drive unit 144Bto shift the zoom lens 14B in the wide angle direction.

After that, when the shutter release button 26 is full-pressed, an S2-onsignal is entered into the CPU 110, and the CPU 110 starts photographyand record processing. Thus, image data acquired in response to theS2-on signal are converted into luminance/color differential signals(Y/C signals) by the image signal processing circuit 134, and theconverted signals, after going through a prescribed processing includinggamma correction, are stored into the memory 120.

The image data stored on the memory 120, after being compressed into aprescribed format (e.g. the JPEG format) by a compressing/expandingcircuit 146, are recorded onto a recording medium 150 via a mediumcontroller 148. When the continuous shooting mode is selected, inresponse to the full pressing of the shutter release button 26, the CPU110 controls each unit so as to acquire a prescribed number of stillimages at prescribed intervals.

If the playback mode is selected with the mode selector switch 50, imagedata (the image data last recorded on the recording medium 150) are readout of the recording medium 150. The read-out image data, after beingexpanded into non-compressed YC signals by the compressing/expandingcircuit 146, are supplied to the monitor 30 via the image signalprocessing circuit 134, the character MIX circuit 135 and the videoencoder 136. In this way, the image recorded in the recording medium 150is reproduced and displayed on the monitor 30.

Frame-by-frame reproduction of an image is accomplished by manipulatingthe cross button 48. If the right side key of the cross button 48 ispressed, the next image data is read out from the recording medium 150,reproduced and displayed on the monitor 30. Or if the left side key ofthe cross button 48 is pressed, the preceding image data will be readout from the recording medium 150, reproduced and displayed on themonitor 30. At this occasion, if sound data is associated with the imagedata of the selected frame, the sound is reproduced in synchronizationwith displaying of the image.

In effect, the digital camera 10 is equivalent to anything that includesthe image pickup function and the recording function as shown in FIG. 3.It is unnecessary for the camera to have an appearance as shown in FIGS.1 and 2. Accordingly, the camera may be a personal computer or a PDAincluding the equivalent configuration of the block shown in FIG. 3.When a personal computer or a PDA does not include the image pickupfunction and the recording function, the image data and the sound datacan be acquired from an external device through a memory or a networksuch as LAN, infrared communication and the Internet.

FIG. 4 is a flow chart of photographing processing according to a firstexemplary embodiment. The processing starts in response to activation ofthe still image shooting mode. This processing is controlled by the CPU110. The program describing the processing is recorded in the ROM 116,and the CPU 110 reads out and executes the program.

In the step P1, it is determined whether or not the shutter releasebutton 26 is half-pressed (S1-on). In response to the determination thatthe button is half-pressed (S1-on, S1 button pressed), the processingadvances to the step P2.

In the step P2, pickup of a sound by the microphone 17 and processingand recording of the picked-up sound by the sound data processingcircuit 149 are started. Instead, the recording of a sound may bestarted immediately in response to activation of the still imageshooting mode.

In the step S3, it is determined whether or not the shutter releasebutton 26 is full-pressed (S2-on). In response to the determination thatthe button is full-pressed (S2-on, S2 button pressed), the processingadvances to the step P4.

In the step P4, operation of acquiring a still image for recording isexecuted, and the acquired image data is temporarily stored on the VRAM122. At this point, the sound recording is continued.

In the step P5, the sound recording is continued for a prescribed timeperiod (e.g. five seconds) after completion of the recording operationof a still image. In response to a lapse of the prescribed time period,the sound recording is terminated.

In the step P6, the CPU 110 instructs a sound analyzing circuit 151 toanalyze the recorded sound. In response to the instruction by the CPU110, the sound analyzing circuit 151 extracts feature data of the sound(sound volume, frequencies, or time series variations in them) recordedin the SDRAM 120 (or the recording medium 150), matches sound featurepatterns (e.g. high or low volume level, long or short speech duration,high or low frequencies) recorded previously in ROM 116 and theextracted feature data with each other, and identifies a sound featurepattern corresponding to the extracted feature data. The identifiedsound feature pattern is then notified to an image effector 152.

In the step P7, the image effector 152 applies an image effectprocessing corresponding to the notified sound feature pattern on thestill image for recording stored on the VRAM 122. In the step P8, theimage on which the effect is applied and the sound stored on the memory120 are recorded in association with each other, on the recording medium150. The original image before the application of the effect may furtherbe associated with them and recorded on the recording medium 150.

For instance, when image data as shown in FIG. 5A is acquired and ascenic pattern of “festival” is notified, contrast enhancement isapplied on the entire image as shown in FIG. 5B, or zooming in whichareas around the perimeter of the image are blurred away and the focusis achieved only on the central area of the image is applied as shown inFIG. 5C in order to cause dynamic atmosphere matching a festival.Instead, processing may be executed such that an original image isshifted to any direction and the shifted image is superimposed on theoriginal image, as shown in FIG. 5D.

Details of the effect may be anything that corresponds to the analyticalresults on the sounds. In addition to the above-mentioned details, anychange in hues, sharpness, distortion, contrast or blurredness may beadopted.

FIGS. 6A to 6D illustrate an example of a correspondence between ananalytical result on a sound and effect processing to be executed by theimage effector 152 in response to notification of the analytical resulton the sound. The details of the effect shown in the figure is anexemplary one. A part or all of them can be combined. Instead ofradially blurring, as a geometrical effect, blurring to the rotationaldirection may be adopted.

When the playback mode is set, the image and the sound recorded inassociation with each other on the recording medium 150 are reproducedin synchronization with each other according to the association. Morespecifically, when reproduction of a desired image is selected, theimage is converted into a video signal and outputted on the monitor 30,and concurrently the sound data associated with the image is convertedinto an analog sound signal and outputted from the loudspeaker 35.

Thus, the sound is recorded during shooting of the still image, aneffect in accordance with the analytical result on the recorded sound isapplied on the still image, and the sound and the still image on whichthe effect is applied are recorded in association with each other,thereby enabling the image to be recorded with realism together with thesound. The still image and sound are reproduced in association with eachother, thereby enabling a viewer to feel an ambience and realism closeto a memory that the photographer felt.

Second Exemplary Embodiment

In the digital camera 10 according to the first exemplary embodiment,following procedures may be taken to preferably record a sound requiredfor feeling a memory.

FIG. 7 is a flow chart of shooting processing according to a secondexemplary embodiment. This processing starts in response to activationof the still image shooting mode. The processing is controlled by theCPU 110. The program describing the processing is recorded in the ROM116, and the CPU 110 reads out and executes the program.

In the step P11, it is determined whether or not the shutter releasebutton 26 is half-pressed. When the button is half-pressed, soundrecording processing in the steps P12 to P18 and image recordingprocessing in the steps P21 to P24 advance concurrently.

In the step P12, it is determined whether or not the sound volumeexceeds the prescribed threshold value Th1 (e.g. 50 phons). When thedetermination is YES, the processing advances to the step P13. When thedetermination is NO, the determination is repeated.

In the step P13, the sound recording starts.

In the step P14, it is determined whether or not the sound volume fallsbelow the prescribed threshold Th1. When the determination is YES, theprocessing advances to the step P15. When the determination is NO, thedetermination is repeated.

In the step P15, it is determined whether or not a duration time “LowTime” exceeds a prescribed threshold Th2 (e.g. ten seconds), concerningthe time when the sound volume falls below the prescribed threshold Th1as a starting time. When the determination is YES, the processingadvances to the step P16. When the determination is NO, thedetermination returns to the step P14.

In the step P16, the sound recording is stopped.

In the step P17, it is determined whether or not a time period “TotalTime” exceeds a prescribed threshold Th3 (e.g. five minutes), concerningthe time when the sound recording starts as a starting time andconcerning the last time when the sound recording stops as an endingtime. When the determination is YES, the processing advances to the stepP18. When the determination is NO, the determination returns to the stepP12.

The step P18 is as with the step P6. The steps P21 to P24 are as withthe steps P3, P4, P7 and P8, respectively. When the “Total Time” exceedsthe prescribed threshold Th3 without the sound volume falling below thethreshold Th1 in the step P14, the processing exits the loop of the stepP14, advances the step P16, and the sound recording is terminated.

FIG. 8 is an example of a timing chart of start/stop of the soundrecording executed according to the sound recording processing in thesteps P12 to P18.

As shown in FIG. 8, at points in time α1, α2, and α3, the sound volumeexceeds the threshold Th1 (step P12: YES), and the sound recordingstarts (step P13).

At each of points in time β1, β2 and β3, the sound volume falls belowthe threshold Th1 (step P14: YES), and then the sound recording isstopped (step P16) at each of subsequent points in time (β1+Th2, β2+Th2and β3+Th2), the points in time when the time period “Low Time” measuredfrom the corresponding points of β1, β2 and β3, during which the soundvolume remains below the threshold Th1, reaches the threshold Th2 (stepP15: YES). If the time period during which the sound volume remainsbelow the threshold Th1 does not reach the threshold Th2, for instancein a time period from γ1 to γ2, the sound recording is not stopped.

The total time period of sound recording “Total Time” exceeds thethreshold Th3, thereby starting the analysis on the sound recordedduring this duration.

Because this can avoid recording in silence to the utmost and theanalyzed feature of the sound becomes easy to figure out, the effect onan image based on the analytical result on a sound can appropriately beapplied.

The timing of the start, stop, restart and termination are notnecessarily determined based on the sound volume. Instead, it may bedetermined based on the frequencies, frequencies and sound, or thetime-series analysis thereon. For instance, the sound recording can bestarted when a sound of a prescribed frequency is detected. Morespecifically, it is said that the frequency of a baby voice is threekHz. When a sound having a frequency of three kHz is detected, it may beassumed that a baby is about to be photographed, and the sound recordingmay be started.

Since the sound recording is thus started, stopped, restarted andterminated according to states of the sounds, only a part necessary torecollect a memory can be recorded, without waste, and the effect on theimage becomes what correctly reflects the state of the sound.

Third Exemplary Embodiment

A variety of effects may be applied on the acquired still image, framesof moving images may be created, and moving images may be created byarranging the frames in time sequence.

For instance, as shown in FIG. 9, an original image I1 is shifted rightor left, upward or downward, or diagonally, and is superimposed on theoriginal image to make image frames L1 and L2. A moving image is createdby alternately arranging a prescribed number (e.g. 100 frames) of theimages L1 that have been shifted left and the images L2 that have beenshifted right. Any recording system, including Motion JPEG, may beadopted to record the moving image.

Instead, as shown in FIG. 10, image frames M1 and M2 may be created byzooming in on a portion having high luminance in an original image I2.

The created moving image data are then recorded on the recording medium150 in association with the sound on the memory 120.

When the moving image is reproduced, the associated sound isconcurrently reproduced, thereby enhancing reality at the time ofshooting.

Fourth Exemplary Embodiment

A pseudo-moving image frame may be created by, when continuous shooting,which is operation acquiring a prescribed count of still image data inprescribed intervals (e.g. continuous three frames in a time interval of0.45 second), is performed, and applying an effect that combines andsuperimposes a part or the whole of the prescribed count of images.

For example, as shown in FIG. 11, when three images J1, J2 and J3 areacquired by continuous shooting, an image K1 is created by superimposingmain subject portions (portions having high luminance) of the images J1and J2, an image K2 is created by superimposing main subject portions(portions having high luminance) of the images J2 and J3, an image K3 iscreated by superimposing main subject portions (portions having highluminance) of the images J3 and J2. The still images K1 to K3continuously show the movement of the subject. The transitional movementof the subject can be grasped, as with a moving image, by reproducingthe images K1 to K3 according to the sequence concerning the shooting ofthe original images. The still images K1 to K3 are stored on therecording medium 150 in association with the sounds. The reproduction ofthe still images K1 to K3 may be transitionally conducted as with aslideshow. At this point, the associated sounds are continuouslyreproduced in the same manner as described above. Thus, the group ofimages showing a continuous movement of the subject is stored inassociation with the sounds, thereby allowing a user to enjoy the imagesand the sounds with realism concerning the shooting.

Fifth Exemplary Embodiment

In the first to fourth exemplary embodiments, it is not necessary torecord the sound picked up by the microphone as it is. Instead, thecontent of the sound may appropriately be edited and recorded togetherwith the image.

For instance, using a digital filter, only a sound in a prescribedfrequency band (e.g. in a case of a human voice: 300 Hz to 3500 Hz) isextracted from the original sound (see FIG. 12A) and recorded. Instead,noise components are eliminated and the sound without noise is recorded.Instead, using ICA (Independent Component Analysis) and so forth, thevoice of a specific person is extracted from the original sound andrecorded (see FIG. 12B). Instead, sound components having a soundwaveform louder than a prescribed value are extracted from the originalsound, and the components are connected together (see FIG. 12C), therebyallowing the amount of data to be reduced and the data to be neatlycontained within the recording time period previously determined.

Sixth Exemplary Embodiment

In the first to fifth exemplary embodiments, an effect may be applied onan image according to analytical results on rotational signals from thegyro sensor 86 and the magnitude of vibration (the shake amount) of thecamera body 12 computed by the shake correction processing circuit 87based on the rotational signals.

For example, as shown in FIG. 13, provided that the shake amounts arecomputed along the time axis, and reference numerals t1, t2, t3, t4 andt5 designate the respective time intervals during which the shake amountgreater than or equal to a prescribed threshold X is computed. If theshutter release button 26 is full-pressed in any one of the timeintervals t1, t2, t3, t4 and t5, the effect is applied on the imageaccording to the shake amount during time interval in which the timingof the full press is involved (full press interval). The parameters ofthe effect are set such that, for instance, in proportion to the shakeamount when full-pressing, a specific portion of the image is enlargedor reduced by an enlargement/reduction ratio the zoom in speed or zoomout speed is changed, the amount of shift is changed, the turning ratiois changed, turning rate is changed and so forth. In FIG. 14, images I2and I3 are created by combining a zooming-in image on the original imageI1 and the original image, thereby expressing dynamic movement with alively feeling corresponding to the situation where the shake occurredduring shooting.

Instead, if the shake amount exceeds the threshold X at least one timein a time period from the half pressing to the full pressing, an effect,for instance, appeal, crawl, strip, slide, dissolve, wipe or the like,may be applied on the image.

The enlargement or reduction ratio of an image, zooming ratio (reductionratio and magnification ratio), amount of shift, shift rate, turningratio, turning speed or the like may be changed according to low/highvariations in sound volume and the rate of variations. An effect, forinstance, appeal, crawl, strip, slide, dissolve, wipe or the like, maybe applied on the image according to the feature of the frequencies ofthe sound.

Various kinds of shooting status detection devices, e.g. a thermalsensor, a pressure sensor etc., may be provided instead of the gyrosensor 86, or in addition to the gyro sensor 86, and an effect isapplied according to the amounts concerning status detected by theshooting status detection devices. For instance, the grip strength onthe camera body 12 is detected by the pressure sensor, and a parameterof effect according to the pressure is set.

Thus, not limited to the sound, an effect is applied on an imageaccording to various kinds of physical quantities that can be detectedfrom the external environment, and an image having reality at the timeof shooting can be created.

Seventh Exemplary Embodiment

In the first to sixth exemplary embodiment, it may be further configuredsuch that a specific movement performed intentionally by thephotographer is detected and a specific effect is applied according tothe physical quantity of the specific movement.

For instance, provided that the photographer momentarily shakes thecamera body 12 after full pressing of the shutter release button 26. Inthis case, as exemplarily shown in FIG. 15, an abrupt variation (pulse)in the shake amount exceeding a threshold Y arises in an interval Taafter full pressing of the shutter release button 26.

When the original image as shown in FIG. 16A is taken in response tofull-pressing of the shutter release button 26, an image M1 is createdby combining a zooming-in image of the original image and the originalimage I1 as shown in FIG. 16B. The image M1 represents the situationwhere the image is photographed with the camera abruptly being shaken,and has reality.

Instead, when the photographer momentarily pushes the camera body 12forward after full-pressing of the shutter release button 26, thedirection of pushing and the amount of pushing are detected by anacceleration sensor or the like. When a prescribed amount of forwardpressing or more is detected, an effect of zooming-in is applied on theoriginal image I1 as shown in FIG. 16C to acquire an image M2. The imageM2 represents the situation where the image is photographed with thecamera being abruptly pressed forward, and has reality.

Instead, when the photographer grips the camera body 12 afterfull-pressing of the shutter release button 26, the grip strength on thecamera body 12 can be detected by the pressure sensor. If the pressurereaches a prescribed threshold (e.g. 10 kg/cm² or more), an effect ofzooming-in may be applied on the original image I1. Instead, when thephotographer's body temperature transferred by grasping the camera body12 by the photographer is detected by the thermal sensor afterfull-pressing of the shutter release button 26 and the temperature ishigher than or equals to a prescribed threshold (e.g. 30 degreescentigrade or more), an effect of turning may be applied on the originalimage I1.

The timing when the photographer makes the digital camera 10 detect thespecific movement may arbitrarily be selected. The timing may be beforeor after full pressing of the shutter release button 26. For instance,when the camera is shaken during reproduction of the original image, animage that is combination of a zooming-in image on the original imageand the original image may be created. Thus, a specific image effect isintentionally created at any point of time.

While examples and embodiments of the present invention have beenexplained in detail, the present invention is not limited to the above,needless to say, various improvements and modifications may be addedwithout departing from the scope of the present invention.

1. An imaging apparatus, comprising: an imaging unit which converts anelectric signal into image data and outputs the image data, the electricsignal being acquired by an image pickup element photoelectricallyconverting light from a subject image-formed through an imaging lens ona photosensitive surface of the image pickup element; a sound acquiringunit which converts a sound around the subject into sound data andoutputs the sound data; a sound analyzing unit which analyzes a featureof the sound based on the sound data outputted from the sound acquiringunit; an effect application unit which applies an effect on the imagedata outputted from the imaging unit, the effect being image processingin accordance with the feature of the sound analyzed by the soundanalyzing unit; and a recording unit which associates the image datagenerated by applying the effect by the effect application unit with thesound data outputted from the sound acquiring unit and records the imagedata and the sound data on a prescribed recording medium.
 2. The imagingapparatus according to claim 1, wherein the effect includes imageprocessing which changes at least one of a hue, sharpness, contrast andblurredness of the image.
 3. The imaging apparatus according to claim 1,wherein the sound analyzing unit extracts a sound volume, a frequency,or a time-series variation thereof as a feature of the sound, comparesthe extracted feature of the sound with a previously-stored featurepattern, and applies the effect which is previously associated with afeature pattern matching the extracted feature of the sound.
 4. Theimaging apparatus according to claim 1, further comprising a soundacquisition control unit which controls a timing of a start, a stop or atermination of conversion and output of the sound data by the soundacquiring unit in accordance with the feature of the sound analyzed bythe sound analyzing unit.
 5. The imaging apparatus according to claim 4,wherein the sound acquisition control unit starts outputting the sounddata when detecting that the volume of the sound becomes larger than orequal to a prescribed value, terminates outputting the sound data whenoutputting the sound data over a prescribed first time period, and stopsoutputting the sound data when the volume of the sound falls equal to orbelow the prescribed value and the falling continues over a prescribedsecond time period.
 6. The imaging apparatus according to claim 1,further comprising a moving image creating unit which creates imageframes of a moving image by applying a plurality of different effects onthe image data outputted from the imaging unit, and creates moving imagedata based on the image frames, wherein the recording unit associatesthe moving image data created by the moving image creating unit and thesound data with each other and records the associated data.
 7. Theimaging apparatus according to claim 1, further comprising a continuousshooting control unit which controls the imaging unit to output aprescribed count of image data at prescribed time intervals; wherein theeffect application unit executes image processing on the prescribedcount of image data outputted from the imaging unit using the image dataand another image data captured before and/or after the image data inaccordance with control of the continuous control unit.
 8. The imagingapparatus according to claim 1, further comprising a sound editing unitwhich edits a content of the sound data from the sound acquiring unit inaccordance with the feature of the sound analyzed by the sound analyzingunit.
 9. The imaging apparatus according to claim 8, wherein the soundediting unit performs at least one of extracting a specific frequencyfrom the sound data, eliminating noise, extracting a sound of a specificperson, and cutting off sound data whose sound volume does not reach aprescribed sound volume.
 10. The imaging apparatus according to claim 1,further comprising a detection unit which detects a physical status ofthe imaging apparatus, wherein the effect application unit applies theeffect in accordance with the physical status detected by the detectionunit.
 11. The imaging apparatus according to claim 10, wherein theeffect application unit applies a specific effect in accordance with aspecific physical status detected by the detection unit.
 12. The imagingapparatus according to claim 1, wherein the effect application unitchanges at least one of a magnification ratio, a zooming speed, anamount of shift, a shift rate, turning ratio, and turning speed in theprocessing of the effect on the image in accordance with a variation ofthe volume of the sound or a rate of variation of the sound.
 13. Areproducing apparatus reproducing image data and sound data inassociation with each other, the image data and sound data recorded onthe prescribed recording medium by the imaging apparatus according toclaim
 1. 14. An imaging method, comprising: converting an electricsignal into image data and outputting the image data, the electricsignal being acquired by an image pickup element photoelectricallyconverting light from a subject image-formed through an imaging lens ona photosensitive surface of the image pickup element; converting a soundaround the subject into sound data and outputting the sound data;analyzing a feature of the sound based on the outputted sound data;applying an effect on the outputted image data, the effect being imageprocessing in accordance with the analyzed feature of the sound; andassociating the image data generated by applying the effect with theoutputted sound data and recording the image data and the sound data ona prescribed recording medium.
 15. A reproducing method reproducing theimage data and the sound data in association with each other, the imagedata and sound data recorded on the prescribed recording medium by theimaging method according to claim
 14. 16. A recording medium on which aprogram is recorded, the program comprising computer-executableinstructions of: acquiring image data and sound data recordedconcurrently with the image data; analyzing a feature of the sound basedon the sound data; applying an effect on the image data, the effectbeing image processing in accordance with the analyzed feature of thesound; and reproducing image data generated by applying the effect andthe sound data in association with each other.